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ABSTRACT 

A study was done of faculty perceptions of the 
accuracy and validity of various time-related personnel activity and 
productivity reporting measures, and of faculty's willingness to 
participate in these measures. In the first stage of the study 1,000 
randomly-selected faculty from 200 institutions of higher education 
that have the highest levels of total separately budgeted 
science/engineer ing research dnd development expenditures were mailed 
a survey instrument entitled "Faculty Perceptions of Activity and 
Productivity Reporting." Faculty responses included suggestions for 
activity and productivity measures and those suggestions appeared in 
a followup survey, "Faculty Perceptions of Measures of Productivity," 
sent to the original cohort and an additional 10,000 science and 
engineering research and development faculty. The first survey 
received a 24 percent response rate; the second a 21 percent response 
rate. Results indicated that faculty think that reporting the 
percentage of time spent in various activities or the number of hours 
in the classroom is not a valid measure of their productivity. 
However, reliance on time-based reporting systems must prevail until 
institutions of higher education and faculty become accountable to 
their sponsors and begin to develop, implement, and make external and 
internal reports based on measures of faculty productivity, (includes 
13 references.) (JB) 
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Abstract 

There has been little insistence that expenditures of federal, state or 
private funds, or that the conduct of instruction, research or service, should 
be related to measures of faculty productivity. 

The faculty's reluctance to participate in current reporting measures is 
partially because they do not feel they are accurate or valid measures. Prior 
to this study, no one had bothered to ask the faculty what they considered to 
be valid measures. 

Measures that surveyed faculty rate as most valid are not always those that 
the Federal and state governments rely upon, or that are most commonly cited 
in the literature. 



ERLC 



4 



Faculty Perceptions of Productivity Measures 

3 

University faculty and administrators have not wanted to concern themselves 
with accountability issues. They have complained over the years concerning the 
inaccuracy and nonvalidity of, and the bureaucracy and paperwork associated 
with the federally required, time-based Personnel Activity Report-'ng, yet they 
themselves have not attempted to identify or implement more appropriate 
measures of accor Ability. 

The concept of "faculty load," or tie amount of work a faculty member does, 
has dominated discussions in academia of faculty productivity since World War 
II. By the late 1950s, many institutions, for administrative and fiscal 
reasons, were attempting to measure faculty load through various methods. 

The concept of Faculty Activity Analyses (FAA), or systems developed to 
record and report faculty efforts and activities for operational analysis, was 
developed in the 1950s in an attempt to answer legislative requests, equalize 
faculty loads, develop unit cost measures and develop common measures of 
faculty activity. By the 1960s, increasingly complex budgets, legislative 
demands for substantiation of funding requests, and federal and state audits 
created an environment in which more universities were undertaking FAA's in an 
attempt to determine how the faculty were spending their time {Swann, 
Saunders, Simpson & Wool ley, 1977). 

Since FAA systems were developed to meet the unique needs of each 
university, all of this uncertainty and lack of consistency lead Ingster 
(1977) to note that the attempt to define productivity for faculty is "almost 
hopeless," and he charged that criteria such as student/faculty ratios and 
weekly hours of work are not significant indicators of levels of productivity. 

A general consensus of the output measures necessary for the conduct of 
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these evaluations is not readily available. Studies in the areas of teaching, 
service and administration are practically non-existent. Even in the area of 
research productivity output measures^ Creswell (1985) indicated that: 
the measures of research productivity in empirical investigations are 
excessively narrow. Beyond publication and citation counts, researchers 
employ few measures. Yes, the criteria used to assess research 
performance vary widely from one type of institution to another. 
Empirical studies should examine broader measures of research (e.g., 
grants obtained, patents, creative projects, and others) and determine 
the correlates that have positive predictive influence (p. 55). 
By 1989, it was noted that little attention had been given to the 
development of indicators of faculty performance (Kurz, Mueller, Gibbons & 
Dicataldo, 1989). Rebne (1990) later advised that since there is considerable 
evidence that occupations differ in forms of output, productivity measures 
should not be restricted to a single channel such as journal articles. 
However, he added "the literature has yet to produce a universally accepted 
measure of research performance" (p. 31). 

Biglan (1973) ""oted that it was inadvisable to collect data on an 
organizational basis, rather than considering the variety of individual 
academic fields. Collection on an organizational basis had a tendency to mask 
the differences among the different academic areas. He also noted that 
studies restricted to one or a few academic areas were not general izable to 
dissimilar academic areas. Biglan concluded that university-wide standards 
for the evaluation of faculty would not be possible. 

Biglan (1973) and Rebne (1986) concluded that since there is a considerable 
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amount of evidence that the fields of study differ in their forms of output, 
productivity measures should not be restricted to a single publishing channel 
such as journal articles. However, the productivity measures that they 
suggested books, monographs, technical reports, and dissertations sponsored 

are relatively limited, and do not include a variety of potential measures 
of faculty productivity in the areas of instruction, research, service and 
administration. 

Productivity is commonly defined as a ratio of output to input, Kopelman 
(1986) noted that in practice, most studies have measured the level of output 
rather than productivity. However, before productivity can be measured, valid 
measures of output and their relevant input measures must be identified. 

Methodology 

The population of the first part of the study consisted of faculty from the 
200 institutions of higher education that have the highest levels of total 
separately budgeted science/engineering research and development expenditures, 
as reported in the National Science Foundation's (NSF) Surveys of Sciences 
Resources Series (1989), One thousand randomly-selected faculty from these 
institutions were sent a survey instrument entitled "Faculty Perceptions of 
Activity and Productivity Reporting" to obtain faculty perceptions of the 
accuracy and validity of, their willingness to participate in various time- 
related personnel activity reporting and productivity reporting systems, and 
to obtain their recommendations for measures of faculty productivity. 

The faculty suggestions for methods of accounting for productivity, along 
with procedures that are currently being used by the same set of 200 
institutions (obtained in a separate study) were incorporated into a follow-up 
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survey instrument entitled "Faculty Perceptions of Measures of Productivity". 
The original sample of 1,000 faculty, as well as an additional 1,000 randomly- 
selected faculty from institutions that ranked 201-400 in separately budgeted 
science/engineering research and development expenditures as reported in the 
NSF Surveys of Sciences Resources Series (1989), were asked to indicate their 
opinions of the validity of reporting faculty productivity on 203 measures of 
productivity for instruction (57 measures), research (74 measures), service 
(39 measures) and administration (33 measures). 

The second survey instrument contains questions that result in data that is 
interval in nature. Means and standard deviations were calculated to 
determine the responding faculty's opin-^ons regarding the validity of 203 
measures of faculty productivity in the areas of instruction, 
research/creative activity, service and administration. The average measures 
of validity for productivity rated by responding faculty, were ranked for each 
of the functional categories of instruction, research, service and 
administration. In addition, measures of productivity that were rated among 
the top five valid measures for instruction, research/creative activity, 
service and administration productivity by either: 1) faculty that were funded 
or unfunded; 2) faculty from institutions that ranked 1-200 or 201-400; or 3) 
academic area, also were identified. 

Results 

Survey Responses - Faculty Perceptions of Activity and Productivity Reporting 
The response rate from the 1,000 faculty from the top 200 institutions that 

were sent the first survey instrument was 24%, 

Accuracy of after-the-fact activity reporting based on percentage of time , 
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On a scale of 1 through 6, with 1 being highly inaccurate and 6 highly 
accurate, responding faculty rate an after-the-fact self-report of their 
activity based on percentage of time between slightly and moderately accurate 
(4.54). Faculty opinions regarding the accuracy of their activity being 
reported after-the-fact based on percentage of time by other representatives 
of the institution were less highly rated. Those representatives, in order of 
level of reporting accuracy include the program director (3.50), department 
chair (3.33), department staff (3.15), dean's staff (2.40) and central 
administration (2.12). The faculty apparently do not perceive the 
institutions' tendency to have their time reported by other persons (Cooper, 
1991), in order to comply with Personnel activity reporting (PAR) as required 
by the Federal government through the U.S. Office of Management and Budget's 
(0MB) Circular A-21. Cost Principles for Educational Institutions (1982), as 
an accurate reporting measure. 

Validity of activity reporting b ased on percentage of time . Responding 
faculty do not regard reporting the percentage of time a faculty member spends 
in each work activity area as a very valid means for demonstrating to sponsors 
that their money was spent for the intended purposes. On a scale of 1 through 
6, with 1 being highly invalid and 6 highly valid, responding faculty, on the 
average, rated this measure as lower than slightly valid (3.87). 

Maximum level of time-rglated specificity . If faculty were required to 
keep a self-report of the time they spend on activities in order to receive 
funding from external sources, they are not willing to do it on a very 
specific level. Of the responding faculty, 23.5 percent indicated that the 
maximum level of specificity that they were willing to provide was an academic 
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term basis. Other respondents indicated that the maximum level of sr^cificity 
that they were willing to provide, in order of frequency, included monthly 
(21.2 percent), weekly (19,0 percent), daily (10.6 percent), annually (9.7 
percent), and hourly (7.9 percent). Only 0.5 p.>rcent of the responding 
faculty was willing to provide a self-report of time on a 15 minute basis, a 
basis that is commonly required of professionals billing out their time in 
private industry. An additional 7.8 percent of the responding faculty 
indirqited that they were unwilling to document their activity in order to 
receive funding from external sources. 

Accuracy of reporting productivitv on measures other than time . Responding 
faculty regard the reporting of faculty productivity based on objective work 
measures other than time as slightly accurate (4.21). 

Validity of reporting productivity on measures other than time . Responding 
faculty rated the validity of reporting faculty productivity based on 
objective work measures other than time slightly higher than the rating they 
had given to the validity of reporting time spent in each work activity area. 
On the average, responding faculty rated reporting faculty productivity on 
objective work measures other than time above slightly valid (4.27). 

Maximum level of productivity reporting . If faculty were required to keep 
a self-report of productivity based on objective work measures in order to 
receive funding from external sources, they do not want to report those 
measures very frequently. Of the responding faculty, 27.6 percent indicated 
the maximum level of that frequency they would be willing to report this 
productivity information was each academic term, and 22.6 percent would be 
willing to report no more frequently than on an annual basis. While the 
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desired infrequency of these reports may be partially due to the faculty's 
desire to reduce the amount of administration and paperwork associated with 
reporting activities, the level of frequency also may be partially due to the 
fact that, in many instances, it takes at least an academic term or more to 
see any real results or products from the faculty's efforts. 

Other respondents indicated the maximum level of frequency they would be 
willing to report this productivity information was, quarterly (11.5 percent), 
semi-annually (11.1 percent), monthly (11.1 percent), and upon project 
completion. An additional 7.4 percent of the responding faculty indicated 
that they were unwilling to document their productivity in order to receive 
funding from external sources. 

Survey Responses - Faculty Perceptions of Measures of Productivity 

The response rate from the 2,000 faculty from the top 400 institutions that 
were sent the second survey instrument was 21%. Of the responding faculty, the 
breakdown by the various areas of study is as follows: 

1) faculty's externally funded project: funded - 49%; unfunded - 51%; 

2) faculty's institutional funding rank: 1-200 - 52%; 201-400 - 48%; 

3) faculty's academic area: Agriculture - 6%; Arts and Humanities - 23%; 
Business and Law - 8%; Education and Home Economics - 12%; Engineering - 
10%; Physical Sciences and Math - 19%; Medical - 12%; and Social Sciences - 
9%.. 

Responding faculty do not, on the average, rate the reporting of 
productivity based on any one measure as highly valid. On a scale of 1 
through 6, with 1 being highly invalid and 6 highly valid, responding faculty, 
on the average, rated the following among the top five in terms of valid 
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measures of productivity for each of the functional categories of instruction, 
research, service and administration: 

Instruction: # of new courses developed (4.83); increase in students' 
subject knowledge (4.81); # of programs/curricula developed (4.75); relevancy 
and currency of syllabi and materials (4.72); # of chaired theses/ 
dissertat /ons completed (4.71); and # of new teaching techniques developed 
(4.71) (see Table 1). 

Other measures that are rated among the top five valid measures of 
instructional productivity by either unfunded or funded faculty, faculty from 
institutions that ranked 1-200 or 201-400, or faculty from different academic 
areas include: # of theses/dissertations chaired; # of doctoral students 
instructed; # of curriculum innovations; # of courses significantly 
restructured; and success of students at next level (courses/work) (see Tables 
1 and 2). 

Research/creative activities: new knowledge created/problems solved 
(5.16); impact of research on discipline (5.15); # of refereed articles 
published (4.94); quality of outlet (4.91); and # of grants awarded (4.80) 
(see Table 1). The tendency to emphasize the number of refereed articles 
published in most productivity studies may be flawed. However, it may, at 
this time, remain as one of the simplest measures to document and compare. 

Other measures that are rated among the top five valid measures of 
research/creative activity productivity by either unfunded or funded faculty, 
faculty from institutions that ranked 1-200 or 201-400, or faculty from 
different academic areas include: # of books published; # of monographs 
published; holder of distinguished chairs; # of patents issued; # of patents 
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commercialized; and # of scholarly awards and honors received (see Tables 1 
and 2), 

Service: # of journal editorships (4.80); impact of the accomplishments 
(4.69); # of national events/conferences organized (4.48); # of reviews/ 
jurying of publications/performances (4.43); and # of national committee/board 
memberships (4.40) (see Table 1). 

Other measures that are rated among the top five valid measures of service 
productivity by either unfunded or funded faculty, faculty from institutions 
that ranked 1-200 or 201-400, or faculty from different academic areas 
include: # of university/college/department conmtittees chaired; # of 
university/college/department committees chaired; # of professional 
committees/boards chaired; provision of service to practitioners; and # of 
technical assistances to business/government/individuals (see Tables 1 and 2). 

Administration: level of leadership provided (5.02); morale of organization 
(4.78); level of achievement (4.76); effective budget management (4.74); and 
reputation of organization (4.67) (see Table 1). 

Other measures that are rated among the top five valid measures of 
administration productivity by either unfunded or funded faculty, faculty from 
institutions that ranked 1-200 or 201-400, or faculty from different academic 
areas include: evaluations by faculty; impact of administration on unit; cost 
effectiveness; # of accomplishments; # of problems solved; level of 
advancement over time; and level of support provided (see Tables 1 and 2). 

Summary, Conclusions and Recommendations 

Reporting the percentage of time the faculty spend in various activities, 
or the number of hours the faculty spend in the classroom, masks the true 
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accomplishments of the faculty. The reliance upon time-based reporting systems 
prevails despite the fact that no study has demonstrated the direct 
correlation between these reporting measures and faculty productivity. 
However, unless institutions of higher education and faculty b come 
accountable to their sponsors, and begin to develop, implement, and make 
external, as well as internal, reports based on measures of faculty 
productivity, they will continue to have these invalid time-based measures 
imposed upon them by external sources. 

Faculty, in particular, must change their attitude toward a serious 
reporting to the public of what they are doing. They are no longer in an 
"ivory tower" where they have to be accountable to no one but themselves. 
Their research and instructional activities have become, over the years, 
heavily based upon the foundation of public support. Reporting their findings 
or results to their colleagues in a journal or at a conference is not enough. 
If the faculty expect the public to continue to support their activities, the 
faculty must accept the fact that they must be accountable to the public for 
those activities. 

Despite the fact that there is an explicit need to have a major change in 
the faculty's attitude regarding the necessity of reporting of their 
productivity, it will be very difficult for administrators to convince the 
faculty to record measures of productivity accurately and to report it in time 
periods that will allow the results to be accurate and valid. If faculty 
expect external funding sources to continue to provide the financial 
foundation for their activities, they must, in turn, be willing to give 
something tangible back to their supporters. 
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Part of the faculty's reluctance to participate in time-based reporting 
measures is that they do not find them to be accurate or valid measures. 
Prior to this study, no one had bothered to ask the faculty what they consider 
to be valid measures of productivity. This study specifically asked the 
faculty to identify those methods of accounting for faculty productivity that 
they felt were better than time-based measures at demonstrating to sponsors 
that their money was spent for the intended purposes, and then assess the 
validity of these various measures for accounting for faculty productivity. 
The results of the second part of this study, which presents the opinions of 
more than 400 faculty, provide the higher education community with a set of 
opinions related to measures of faculty productivity that could provide the 
relevant philosophical base related to a new set of premises, that will allow 
information gathering related to valid and acceptable measures of 
productivity, and that could be provided to sponsors and the general public to 
demonstrate that their funds have been well spent and used for their intended 
purposes. 

In the further development of the internal/external productivity reporting 
systems, the faculty's opinions on what they feel are valid measures of 
productivity for their own jobs should be given serious consideration by 
university administrators, sponsors and public officials mandating such 
reporting. 

It is apparent from the results of the second survey, "Faculty Perceptions 
of Measures of Productivity", that faculty do not, on the average perceive any 
single output measure as highly valid. This may be due to the fact no single 
measure is rated as highly valid by all respondents, but it also could be due 
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partially to the fact that several faculty noted that no single measure is a 
good indicator of faculty productivity. Faculty have a multitude of 
responsibilities within each of the functions of instruction, research, 
service and administration, and limiting the analysis of their productivity to 
a few measures would not present a full picture of their efforts. 

The tendency to emphasize the number of refereed articles published in most 
productivity studies may be flawed. However, it may, at this time, remain as 
one of the simplest measures to document and compare. Some of the output 
measures rated as highly valid provided by the faculty, such as impact of the 
research on the discipline or level of leadership, would be extremely 
difficult to document, quantify or evaluate. Further research is needed to 
determine ways to document, quantify or measure some of these other measures- 
Methods for taking into account the quality of the output, as well as 
quantity, also should be developed. 

It is recommended that the second survey instrument, "Faculty Perceptions 
of Measures of Productivity," be redistributed to faculty from other types of 
institutions of higher education. It is very likely that these output measures 
could be rated differently by faculty at other types of institutions of higher 
education that have less of a focus on research and more of an 
instructional/service orientation. Before these output measures could be 
applied to such instructional/service-oriented institutions, the study would 
have to be replicated at these institutions. 

University administrators and external sponsors also should be surveyed to 
determine if their expectations for faculty output are related to, or in 
opposition with, faculty intentions. Other assessment constituencies, such as 
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accrediting agencies, coordinating boards, regents, parents and students, also 
could be surveyed for their impressions of these measures of faculty 
productivity. 

If a more universally accepted set of output measures can be identified, 
these measures should be used to: 

1. document productivity by correlating these output measures to relevant 
input measures. This will require the additional analysis of which 
input measures are relevant and should be correlated with what output 
measures. 

2. test whether or not there is indeed a correlation between productivity 
in terms of outputs produced per unit of time* 

3. attempt to determine the relevant worth of one unit of output in one 
area versus another (i.e., is one refereed journal article equivalent to 
two non-refereed journal articles or the instruction of a class of 30 
students). This "relevant worth" may need to vary across disciplines 
and institutions. 

In 1929, Reeves and Russell concluded that the evaluation of "faculty load" 
was an extremely difficult problem, that existing measures of faculty 
productivity were unsatisfactory and incomplete, but that some measure must be 
employed. Thirty years later, Stickler (1960) conceded that the situation had 
not changed. Unfortunately, another thirty years has passed, with little 
advancement. This study has been an att. ;pt to rectify that problem by 
presenting the faculty^s perceptions of various measures of faculty 
productivity. 
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